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IPSATIVE  RANKINGS  AS  AN  INDICATOR  OF  JOB-WORKER  MATCH 


I.  INTRODUCTION 


Ratings  have  been  used  as  a criterion  in  so  many  prediction  studies  that  it  is  pointless  to  reference 
even  a typical  few.  Ratings,  in  one  form  or  another,  are  involved  in  almost  every  personnel  decision,  and  yet 
we  know  that  rating  scores-particularly  those  used  operationally -are  beset  by  difficulty  and  error  at  every 
turn.  One  of  the  moat  commonly  recognized  rating  errors  is  that  which  is  called  “leniency”  error,  and  it 
refers  to  the  common  observation  that  different  raters  have  different  standards  of  excellence  and  that  much 
of  the  variance  in  a set  of  rating  scores  is  due  to  simple  variability  of  the  means  of  ratings  given  by  various 
raters.  This  is  the  component  of  the  rating  score  which  Cronbach  (1955)  called  “Elevation”  and  which 
“. . .reflect  his  (the  rater’s)  way  of  using  the  response  scale.” 

When  ratings  are  given  operationally  in  a setting  which  requires  that  the  ratings  be  accessible  to  the 
ratee,  the  ratings  typically  become  inflated  so  that  the  rating  scores  cluster  around  the  maximum  score,  and 
variance  is  sharply  diminished.  This  effect  probably  occurs  because  ratings  are  usually  made  by  supervisors, 
who  are  faced  with  conflicting  demands-they  are  required  to  evaluate  their  employees  but,  for  the  sake  of 
production,  they  must  be  careful  not  to  demotivate  their  employees  with  low  ratings.  Except  for  the 
obvious  solutions  to  this  dilemma  (taking  the  evaluative  duty  away  from  the  supervisor  or  allowing  the 
ratings  to  be  made  privately,  neither  of  which  is  likely  to  be  acceptable),  methods  for  collecting  useful 
rating  data  in  a public  rating  environment  are  not  numerous. 

A way  to  prevent  inflation  in  ratings  is  to  require  the  rater  to  rank-order  elements  of  a list  of 
pertinent  characteristics  from  strongest  to  weakest  as  they  are  observed  in  the  ratee.  This  procedure  will 
provide  a scoring  of  these  characteristics  which  is  ipsative  in  nature-that  is,  the  profile  issuing  from  this 
exercise  provides  information  about  the  ratee  only  as  his  characteristics  are  compared  with  each  other.  The 
scores  are  not  necessarily  comparable  from  one  ratee  to  the  next,  as  are  normative1  scores,  because  there  is 
no  common  basis  of  comparison.  The  weakest  characteristic  of  Person  A can  conceivably  be  better  than  the 
best  characteristic  of  Person  B,  so  direct  comparisons  across  ratees  of  these  ipsative  rankings  are  not 
feasible. 

However,  rankings  of  a set  of  characteristics  can  be  made  for  both  a person  and  a job  (e.g.,  “How 
would  you  rank  these  characteristics  for  an  ideal  incumbent?”  or  “Now  rank  these  characteristics  to  show 
how  important  they  are  to  successful  performsnce.”),  and  the  two  rankings  can  then  be  compared  through 
a rank-order  correlation  coefficient.  The  coefficient  produced  should  reflect  a kind  of  worker-job  match,  at 
least  in  terms  of  patterns  of  relative  importance  of  characteristics.  A useful  quality  of  this  coefficient  is  that 
one  is  produced  for  each  ratee,  and  it  may  then  be  considered  as  another  variable  (albeit  one  with  an 
unusual  distribution),  comparable  across  subjects.  It  should  be  kept  in  mtod  that  this  coefficient  of 
worker-job  match  is  different  from  the  more  typical  normative  rating  score  in  that  nothing  in  this  score 
indicates  an  overall  level  of  an  employee’s  excellence.  Only  the  relative  strengths  of  the  items  on  the  list  of 
characteristics  is  displayed  in  the  metric,  and  individual  differences  arise  solely  from  variance  in  degree  of 
match  between  the  two  rankings.  This  study  is  an  attempt  to  investigate  the  utility  of  ipsative  rankings  as  a 
criterion  metric  when  used  alone  and  in  combination  with  the  more  typical  normative  ratings. 

Every  worker  in  criterion  research  is  wen  aware  of  the  conceptual  and  methodological  problems 
involved.  Criterion  measures  are  difficult  to  devise,  but  a design  which  can  demonstrate  the  adequacy  of  a 
criterion  metric  is  even  more  difficult  to  conceive.  One  approach  is  to  find  a situation  with  an  acceptable 


For  ■ more  complete  description  of  the  ipeative  n.  normative  concept,  tee  CatteO  (1944,  1957). 


ultimate  criterion  against  which  one  can  validate. the  criterion  metric  under  consideration.  “Validating”  a 
criterion  makes  practical  sense  only  in  certain  situations,  such  as  (a)  one  which  involves  an  attempt  to 
replace  a very  expensive  measurement  with  one  which  is  less  expensive,  or  (b)  one  in  which  an  ultimate 
criterion  may  be  collected  only  once,  for  the  purpose  of  attempting  to  evaluate  the  extent  that  another 
routinely  collectible  criterion  will  serve  as  a good  substitute,  or  (c)  a situation  such  as  this,  where  the 
particularjjriterion  is  of  interest  only  in  studying  a method  of  collecting  data  which  might  be  transferable 

to  "other  situations  where  a more  ultimate  criterion  might  not  be  readily  available.  This  study,  although  it 

occurs  in  a specific  training  situation,  is  a study  of  method. 


II.  METHODOLOGY 

The  subjects  of  this  study  were  trainees  in  each  of  three  Air  Force  technical  schools. 

a.  Security  Specialist  (L3ABR81 130)  N * 1 12,  School  A. 

b.  Ground  Radio  Equipment  Repairman  (E3ABR30434)  N = 68,  School  B. 

c.  Jet  Engine  Mechanic  (3ABR42632)  N = 94,  School  C. 

Those  three  schools  were  selected  primarily  because  they  still  provide  final  school  grades  upon 
graduation  of  the  students.  School  grades  provided  an  essential  external  criterion.  Each  student  was 
required  first  to  provide  a normative  rating  for  each  of  his  peers  on  a 10-point  scale  from  1 (“Inferior”)  to 
10  (“Superior”)  based  on  how  well  they  were  “doing  in  this  course.”  The  same  normative  rating  was  also 
collected  from  each  instructor  on  each  of  his  students. 

Additionally,  each  student  was  required  to  provide  a rank  ordering  of  a list  of  characteristics  (see 
Appendix  A)  for  each  of  his  peers.  He  was  to  consider  one  pe<r  at  a time  and  select  which  trait  was 
strongest  in  the  peer,  which  was  second  strongest,  and  so  on  through  the  list  of  10  characteristics.  Then  the 
student  was  required  to  rank  the  same  10  characteristics  “to  show  how  important  they  are  to  the  successful 
completion  of  this  course.” 

The  instructor  of  each  group  was  also  required  to  rank  the  characteristics  for  each  student  and  for  the 
course.  The  instructor’s  supervisor  provided  a ranking  of  the  characteristics  for  the  course. 

It  is  believed  that  the  selection  of  a particular  set  of  traits  is  not  crucial  in  this  context,  so  long  as 
some  of  the  traits  seem  relevant  to  job  performance.  The  rater  is  required  only  to  rank  each  trait  relative  to 
the  other  traits,  as  they  appear  in  the  ratee,  and  as  they  are  required  by  a particular  job.  If  some  are  not 
highly  job  relevant,  they  should  tend  to  be  placed  at  the  lower  end  of  the  rank  order  for  that  job-although 
for  some  other  job  they  may  be  near  the  top. 

It  was  intended  that  data  be  collected  from  10-member  rating  groups  with  306  cases.  Because  of 
various  problems  with  missing  data,  the  N of  306  dropped  to  176  matched  students  with  complete 

information,  and  it  was  with  this  sample  that  the  first  analyses  were  made.  Distributions  indicated  that  109 

instructors  had  failed  to  evaluate  their  students,  either  normatively  or  ipsatively,  and  that  98  of  these  cases 
could  be  saved  if  the  instructor  rating  variables  were  dropped.  This  was  done,  and  the  later  analyses  were 
made  on  274  cases. 

At  this  point,  there  were  opportunities  to  calculate  nine  rank-order  correlations  (rho)  for  each 
subject: 

1 . Course  ranking  by  students  vs.  student  ranking  by  students. 

2.  Course  ranking  by  students  vs.  student  ranking  by  instructor. 

3.  Course  ranking  by  students  vs.  student  self-ranking. 

4.  Course  ranking  by  instructor  vs.  student  ranking  by  students. 

5.  Course  ranking  by  Instructor  vs.  student  ranking  by  instructor. 

6.  Course  ranking  by  instructor  vs.  student  self-ranking. 


7.  Course  ranking  by  supervisor  vs.  student  ranking  by  students. 

8.  Course  ranking  by  supervisor  vs.  student  ranking  by  instructor. 

9.  Course  ranking  by  supervisor  vs.  student  self-ranking. 

These  values  were  computed  to  provide  evidence  concerning  the  relative  quality  of  rankings  from 
different  sources.  The  school  setting  in  which  this  study  was  performed  contained  desirable  analogs  to 
on-the-job  environments-the  instructor  was  analogous  to  a first-line  supervisor,  the  instructor’s  supervisor 
was  analogous  to  a work -situation  second-line  supervisor,  and  the  school  grades  were  analogous  to  a “real” 
on-the-job  criterion  that  is  rarely  available  in  the  real  world.  Since  we  know  that  operational  normative 
ratings  are  influenced  to  some  degree  by  a supervisor’s  tendency  to  over-rate  his  workers,  it  might  be  better 
in  the  projected  use  of  a worker-job  match  index  to  have  the  qualities  ranked  for  the  job  by  someone  other 
than  whoever  ranks  the  qualities  in  the  workers.  This  refinement  would  make  it  considerably  more  difficult 
for  the  supervisor -or  anyone  else-to  manipulate  the  evaluation  system. 

Ultimately,  there  are  two  crucial  questions  about  the  job-worker  match  coefficient. 

1.  When  final  school  grades  are  predicted  by  normative  ratings,  does  the  addition  of  the  job-worker 
match  coefficient  increase  prediction  when  it  is  added  to  the  prediction  system?  This  is  another  way  of 
asking  whether  the  job-worker  coefficient  contains  unique  valid  variance.  If  so,  the  indication  would  be  that 
both  normative  and  ipsative  rating  data  should  be  collected  for  use  as  a rating  criterion.  If  not,  the  next 
question  becomes  important. 

2.  When  final  school  grades  are  predicted  by  job-worker  match  coefficients,  does  it  improve 
prediction  to  add  normative  ratings  io  the  system?  This  question  is  important  because  of  the  possibility 
that  neither  a normative  nor  an  ipsative  rating  adds  unique  valid  variance  to  the  other,  but  each  predicts  the 
criterion.  It  is  conceivable  that  both  kinds  of  ratings  measure  essentially  the  same  thing.  If  this  should  be 
the  case,  data  such  as  the  job-worker  match  coefficient  would  have  a substantial  practical  advantage  over 
ordinary  normative  ratings  because  they  cannot  be  so  easily  manipulated.  If  the  job-worker  coefficient  does 
not  add  unique  valid  variance,  and  if  normative  ratings  do  add  such  variance,  then  it  may  safely  be  said  that 
the  job-worker  match  data  collected  in  this  instance  were  not  of  value. 


m.  RESULTS  AND  DISCUSSION 

Nine  rank-order  correlations  were  computed  for  each  student,  forming  nine  new  variables. 
Correlations  among  these  nine  variables,  the  global  rating  from  the  instructor,  the  global  peer  ratings 
averaged  across  groups,  each  subject’s  self  rating,  and  school  grades  are  shown  for  each  school  in  Tables  1 , 
2,  and  3.  There  were  severe  problems  with  the  data,  so  that  when  all  cases  with  complete  records  were 
assembled,  N’s  had  shrunk  to  84,  33,  and  59  for  the  three  schools.  Nevertheless,  there  were  some  obvious 
trends  in  the  matrices. 

1.  Clearly  the  highest  validities  against  final  school  grades  among  the  three  normative  ratings  (by 
instructors,  by  self,  and  peer  average)  occurred  for  peer  ratings  in  each  of  the  three  schools.  Average 
correlations  with  final  school  grade  across  the  three  schools  were  .19  for  instructor  ratings,  .20  for  self 
ratings,  and  .45  for  average  peer  rating*  (Table  4).  This  result  argues  for  the  superiority  of  peer  ratings 
among  sources  of  normative  rating  data,  despite  the  fact  that,  among  the  three  sources,  peer  ratings 
uniformly  had  much  the  smrflest  standard  deviations  in  all  three  schools. 

2.  Inspecting  the  sub-matrix  which  displays  the  intercorrelations  among  the  job-worker  coefficients, 
one  is  struck  by  the  size  of  the  correlations  between  those  arising  from  the  same  source  of  student 
evaluation,  regardless  of  the  genesis  of  the  job  evaluation  component.  For  example,  in  Table  I,  CP-SS 
(courae  rankings  by  peers;  student  ranking  by  students)  correlates  .70  with  CI-SS  (course  ranking  by 
instructor;  student  ranking  by  students).  The  only  other  large  CP-SS  correlation  Is  £3  with  CS-SS  (course 
by  supervisor;  student  by  students).  If  we  may  call  the  first  two  letters  of  each  job-worker  coefficient  a 
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Table  4.  Validities  of  Normative 
Ratings  Against  Final  School  Ratings 

<N  = I 76) 


School 

Normally#  Rating  Scores 

Instructor 

son 

Paar 

1 

.26 

.22 

.45 

2 

.38 

39 

.62 

3 

-.09 

-.02 

22 

Average1 

.19 

.20 

.45 

* Converted  to  Fisher  z’s,  averaged,  and  reconverted  to  r’s. 


“prefix”  and  the  last  two  letters  a “suffix,”  it  is  obvious  that  a strong  determiner  of  the  size  of  the 
relationship  is  similarity  of  suffix.  The  high  correlations  in  this  sub-matrix  are  the  two  already  mentioned 
and  CP-S1  versus  CI-SI  (.77);  CP-SI  versus  CS-SI  (.67);  CP-SR  (course  by  students;  student  self-ranking) 
versus  CI-SR  (.79);  CP-SR  versus  CS-SR  (.64);  CI-SS  versus  CS-SS  (.46);  CI-SI  versus  CS-SI  (.54);  and  CI-SR 
versus  CS-SR  (.49).  The  interesting  point  of  these  correlations  is  that  every  one  of  them  is  between 
coefficients  with  the  same  suffix.  Similarity  of  prefix  seems  to  have  little  or  no  effect  on  the  size  of  the 
correlation.  What  this  seems  to  be  saying  is  that  the  ipsative  ranking  of  the  subjects  is  different  from 
different  sources  (that  is,  whether  the  rankings  are  obtained  from  self,  from  peers,  or  from  the  instructor), 
but  the  ranking  of  the  qualities  in  the  job  (that  is,  school  performance)  must  be  rather  similar  across  the 
three  sources  (the  supervisor,  the  instructor,  and  the  students). 

This  pattern  of  high  correlations  among  similar  suffixes  continues  with  remarkable  regularity  through 
all  three  tables,  except  for  one  exception  in  Table  3.  The  exception  is  that  the  set  of  course-by -supervisor 
job-worker  coefficients  does  not  provide  high  intercorrelations  with  any  variable  at  this  school.  It  appears 
that  these  course  rankings  were  not  very  good  in  any  respect-notice  that  these  three  coefficients  correlated 
very  poorly  with  final  school  grade  and  that  the  means  of  these  numbers  were  uniformly  very  close  to  zero. 

When  validities  for  final  school  grades  yielded  by  similar  prefixes  are  averaged  across  suffix  and  across 
schools,  the  following  results  are  obtained:  CP-XX  = .15;  Cl-XX  = .21;  and  CS-XX  = .08  (Table  5).  This 
indicates  that,  across  all  subjects,  the  best  estimates  of  qualities  in  the  job  are  provided  by  the  instructors 
and  the  least  valid  by  the  supervisors.  When  validities  are  averaged  across  suffixes  and  schools,  XX-SS  = .20, 
XX-SI  = .09,  and  XX-SR  = .15.  The  interpretation  is  that  peers  give  the  best  ipsative  rankings  of  qualities  in 
the  subjects  and  the  instructors  provide  the  least  valid.  Combining  these  two  findings,  it  seems  reasonable  to 
expect  the  highest  final  school  grade  validities  among  the  job-worker  coefficients  to  occur  with  that 
particular  blend  of  rankings  of  qualities  in  the  job  by  instructors  and  ranking  of  qualities  in  the  students  by 
their  peers  (Cl-SS).  This  combination  provides  the  highest  validities  of  the  job-worker  coefficients  averaged 
across  the  three  schools  (.25). 

The  regression  problems  culminating  this  study  posed  a problem.  There  were  many  missing  cases  of 
instructors  rating  students.  Since  regression  problems  require  rather  large  N values  to  produce  stable 
results,  the  largest  possible  N was  desired.  It  was  decided  to  perform  the  regressions  on  problems  involving 
instructor-provided  student  rankings  first  on  the  smaller  N (176)  and  perform  the  other  regressions 
afterward  on  the  larger  N (274).  The  intercorrelation  matrix  recomputed  on  the  larger  N is  not  given  here 
since  there  were  no  major  changes  in  the  pattern  of  intercorrelations.  Table  6 contains  the  information 
from  the  regressions  performed  on  the  instructor  variables,  school  by  school,  and  Table  7 contains  similar 
results  from  those  variables  not  involving  ratings  of  students  provided  by  instructors. 

Table  6 indicates  that,  in  two  of  the  three  schools,  normative  instructor  ratings  predict  final  school 
grades  as  well  as  a set  of  predictors  composed  of  normative  instructor  ratings  plus  job-worker  coefficients 
derived  from  student  rankings  provided  by  instructors.  In  the  third  school,  just  the  reverse  is  true-the 
job-worker  coefficients  provide  all  the  predictive  power  available  from  a set  of  job-worker  coefficients  plus 
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Table  6.  Regression  ProUemi  Against  Final 
School  Grades,  Involving  Instructor 
Rankings  of  Students 

(N- 176) 


Mods** 

R2 

FuN 

n* 

Restr 

School  A (N  - 84) 

Instr  Rtg  + CP-S1 

.07 

Instr  Rtg  alone 

.07 

CP-SI  alone 

.00b 

Instr  Rtg  + Cl -SI 

.07 

Instr  Rtg  alone 

.07 

CI-SI  alone 

.00b 

Instr  Rtg  + CS-SI 

.07 

Instr  Rtg  alone 

.07 

CS-SI  alone 

School  B (N  = 33) 

,00b 

Instr  Rtg  + CP-SI 

.15 

Instr  Rtg  alone 

.15 

CP-SI  alone 

.01 b 

Instr  Rtg  + CI-SI 

.15 

Instr  Rtg  alone 

.15 

CI-SI  alone 

.03 

Instr  Rtg  + CS-SI 

.15 

Instr  Rtg  alone 

.15 

CS-SI  alone 

School  C (N  = 59) 

.00b 

Instr  Rtg  + CP-SI 

.10 

Instr  Rtg  alone 

.01 b 

CP-SI  alone 

.10 

Instr  Rtg  + CI-SI 

r .14 

Instr  Rtg  alone 

.01c 

CI-SI  alone 

.13 

Instr  Rtg  + CS-SI 

(d) 

*CP  * Course  by  peers;  Cl  » Course  by  instructor;  CS  • 
Course  by  supervisor;  SI  * Students  by  instructor. 

^Significantly  different  from  full  model  R*  beyond  .05 
level. 

Significantly  different  from  full  model  R*  beyond  .01 
level. 

“*No  component  of  full  model  significantly  different  from 
zero. 


Table  7.  Regreaion  Problem*  Against  Final 
School  Grade*  Not  Involving  Inatructor 
Ranking*  of  Student* 

(N  -274) 


Modal* 

n* 

Full 

n* 

Raatr 

Peer  Rtg  + CP-SS 

School  A.  (N  = 112) 

.10 

Peer  Rtg 

09k 

CP-SS 

.03b 

Peer  Rtg  + CP-SR 

.10 

Peer  Rtg 

.09 

CP-SR 

.02" 

Peer  Rtg  + Cl-SS 

.11 

Peer  Rtg 

.09. 

CI-SS 

•05b 

Peer  Rtg  + Cl-SR 

.11 

Peer  Rtg 

■09h 

CI-SR 

.03b 

Peer  Rtg  + CS-SS 

.10 

Peer  Rtg 

.09 

CS-SS 

.05  b 

Peer  Rtg  + CS-SR 

.12 

Peer  Rtg 

.09 

CS-SR 

04b 

Peer  Rtg  + CP-SS 

School  B (N  = 68) 

.50 

Peer  Rtg 

.50. 

CP-SS 

•10b 

Peer  Rtg  + CP-SR 

.50 

Peer  Rtg 

.50. 

CP-SR 

,04b 

Peer  Rtg  + CI-SS 

.52 

Peer  Rtg 

.50. 

CI-SS 

■23b 

Peer  Rtg  + CI-SR 

.50 

Peer  Rtg 

.50. 

CI-SR 

,03b 

Peer  Rtg  + CS-SS 

.52 

Peer  Rtg 

JO. 

CS-SS 

.07  b 

Peer  Rtg  + CS-SR 

.50 

Peer  Rtg 

.50. 

CS-SR 

,02b 

Peer  Rtg  + CP-SS 

School  C (N  = 94) 

.01 

Peer  Rtg 

.01 

CP-SS 

.01 

Peer  Rtg  + CP-SR 

.02 

Peer  Rtg 

.01 

CP-SR 

.01 

Peer  Rtg  + CI-SS 

.01 

Peer  Rtg 

.01 

CI-SS 

.00 

Peer  Rtg  + CI-SR 

.01 

Peer  Rtg 

.01 

CI-SR 

.01 

Peer  Rtg  + CS-SS 

.01 

Peer  Rtg 

.01 

CS-SS 

.00 

Peer  Rtg  + CS-SR 

.02 

Peer  Rtg 

.01 

CS-SR 

.01 

*CP  ■ Count  by  peeti;  Cl  * Course  by  instructors;  CS  ■ 
Course  by  supervisor;  SS  • Student  by  Students;  SR.  • Student 
self-rating. 

b Significantly  different  from  full  model  R’  beyond  .01 
level. 
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instructor  ratings.  This  finding  leaves  us  in  an  ambiguous  state,  particularly  when  we  consider  that 
normative  instructor  ratings  are  not  analogous  to  operational  supervisor  ratings  in  one  important  respect. 
There  was  no  pressure  on  these  instructors  to  distort  their  ratings  as  there  is  on  supervisors.  Probably  these 
instructor  ratings  are  considerably  more  accurate  than  those  taken  from  supervisors  in  an  operational 
situation. 

Table  7 presents  results  that  are  much  less  equivocal  than  those  in  Table  6.  In  schools  A and  B,  the 
contribution  of  the  normative  peer  rating  to  the  ipsative  peer  rankings  is  significant  beyond  the  .01  level. 
The  contribution  of  the  ipsative  peer  rankings,  on  the  other  hand,  is  not  significant  when  used  with 
normative  peer  ratings  and  is  sometimes  not  significant  when  used  alone.  In  school  C,  neither  peer  ratings 
nor  the  job-worker  indexes  are  significantly  correlated  with  final  school  grades. 


IV.  SUMMARY  AND  CONCLUSIONS 

As  is  so  often  the  case  with  the  first  study  of  a series  investigating  the  efficacy  of  an  idea,  the  results 
of  this  study  did  not  firmly  establish  the  superiority  of  either  ipsative  or  normative  rankings.  In  some 
combinations,  some  job-worker  match  coefficients  predicted  final  school  grades  better  than  did  normative 
ratings,  either  alone  or  in  combination.  In  most  combinations,  the  reverse  was  true,  but  there  seems  to  be 
enough  promise  in  the  technique  to  warrant  further  investigation. 

In  this  particular  situation,  it  appears  that  the  job-worker  coefficient  issuing  from  job-ranking  by 
instructors  and  student  ranking  by  their  peers  produced  the  best  prediction  of  final  school  grades.  Special 
attention  should  be  given  to  this  metric  in  future  work. 
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APPENDIX  A:  STUDENT  CHARACTERISTICS  RANKING  SHEET 


Below  is  a list  of  10  personal  characteristics  with  their  definitions.  Take  time  now  to  read  the  definitions. 


1.  Carefulness 

2.  Responsiveness 

3.  Initiative 

4.  Creativity 

5.  Tolerance  of  Stress 

6.  Cooperation 

7.  Adaptability 

8.  Writing  Ability 

9.  Speaking  Ability 
10.  Reasoning  Ability 


Taking  the  necessary  time  and  effort  to  produce  work  which  has  few  errors. 

Responding  very  quickly  to  the  demands  of  the  course,  often  completing  work 
sooner  than  required. 

Working  well  alone  with  very  little  guidance,  looking  ahead  to  future  course 
requirements,  and  doing  what  should  be  done  without  being  directed  to  do  so. 

Using  unusual  and  clever  ideas  to  come  up  with  original  solutions  to  problems. 

Working  well  under  pressure; one’s  ability  to  perform  isn’t  hurt  by  tension. 

Working  well  with  others  to  perform  tasks  and  solve  problems. 

One  can  easily  change  what  he  does  to  fit  the  needs  of  the  course. 

Writing  clearly  so  that  people  can  understand. 

Speaking  clearly  so  that  people  can  understand. 

Understanding  the  most  important  ideas  taught  in  the  course,  and  using  these 
ideas  to  perform  tasks  and  solve  problems. 
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